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METHOD AND SYSTEM FOR EVALUATING DOCUMENTS 

FIELD OF THE INVENTION 

The present invention relates to methods and systems for evaluation of 
documents such as written responses. More specifically, the present invention relates to a 
method and system for on-line electronic grading of essays in response to examination 
questions. 



10 



BACKGROUND OF THE INVENTION 

The notion of automated essay grading has been in existence for decades, 

since the 1960 f s when mainframes were in use. Generally, a computerized essay grading 
*;J system relies on "proxies" for writing ability that correlate with previously graded essays. 

*f j i <- For example, for a given essay question, an evaluator would grade a number of responsive 

pi 

J 1 essays and the software would then analyze the correlation between the grade the evaluator 

M| gave to an essay and the structure of the essay, such as the number of paragraphs and the 

"!! average word length. Ponderous mathematical algorithms would be employed to establish 

■a formulae that would be applied to ungraded essays. 

2Q Computerized essay grading has become a trend in standardizing essay 

examination administration. For example, United States Patent No. 5,180,309 discloses an 
JSj automated answer evaluation and scoring system and method for achieving automated 

evaluation and scoring of a participant's response to questions selected pseudo-randomly. A 
plurality of questions are available for each of a plurality of categories, in a question file. 
25 Also, systems employing human readers have been developed to retain 

consistency and accuracy in essay grading. For example, United States Patent No. 
5,987,302 discloses an on-line essay evaluation system offering students the opportunity to 
prepare practice essays and to submit the essays to trained, expert readers, and to retrieve an 
evaluation at the student's convenience. 

30 

SUMMARY OF THE INVENTION 

The present invention is directed to a computer-assisted method of 
evaluating an essay. The computer-assisted method comprises receiving an essay 
concerning an essay topic; electronically comparing textual content of the essay with a first 
35 number of terms related to the essay topic; identifying missing terms among the first 
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number, the missed terms being those terms which are not present in the textual content of 

the essay; and transmitting the missed terms. 

The computer-assisted method may further comprise transmitting at least one 

essay score along with the identified terms. It may also further comprise transmitting at 
5 least one essay subscore along with the identified terms and the essay score. In addition, it 

may further comprise identifying and transmitting terms among the first number which are 

present in the textual content of the essay. 

The present invention is directed to a computer storage medium having 

executable software code stored thereon. The executable software code includes code to 
10 receive an essay concerning an essay topic; code to electronically compare textual content 

of the essay with a first number of terms related to the essay topic; code to identify terms 

among the first number which are not present in the textual content of the essay; and code 

to transmit the identified terms. 

The computer storage medium may further include code to identify and 
15 transmit terms among the first number which are present in the textual content of the essay. 

Also, it may further include code to generate and transmit a score concerning the essay. In 

addition, it may further include code to generate and transmit a subscore concerning the 

essay. 

The present invention is directed to a computer-assisted method of 

20 evaluating an essay on an essay topic. The computer-assisted method comprises providing a 
first essay on the essay topic, wherein the first essay is a model essay; extracting a first set 
of terms found in the model essay; receiving a second essay on the essay topic; 
electronically searching through the second essay for terms within the first set; and 
transmitting those terms within the first set which are not present in the second essay. 

25 The computer-assisted method may further comprise transmitting those 

terms within the first set which are present in the second essay. It may also further comprise 
providing and transmitting a score concerning the second essay. In addition, it may also 
further comprise providing and transmitting a subscore concerning the second essay. 

The present invention is directed to a computer-assisted method of taking an 

30 essay examination. The computer-assisted method comprises drafting an essay about an 
essay topic; submitting the essay for evaluation by a computer; and receiving a first list of 
terms which are related to the essay topic, but were not included in the essay that was 
drafted and submitted for evaluation. 

The computer-assisted method may further comprise receiving a second list 

35 of terms which are related to the essay topic and were included in the essay that was drafted 
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and submitted for evaluation. It may also further comprise receiving a score concerning the 
essay in addition to the first list of terms. It may also further comprise receiving a subscore 
concerning the essay in addition to the first list of terms and the score. In addition, it may 
further comprise revising an existing essay which previously was submitted for evaluation. 
5 The present invention is directed to a computer having associated therewith 

executable software code configured to grade an essay. The executable software code 
includes code to receive an essay concerning an essay topic; code to electronically compare 
textual content of the essay with a first number of terms related to the essay topic; code to 
identify terms among the first number which are not present in the textual content of the 
10 essay; and code to transmit the identified terms. 

The executable software code may further include code to identify and 
transmit terms among the first number which are present in the textual content of the essay. 
„f Also, it may further include code to generate and transmit a score concerning the essay. In 

addition, it may further include code to generate and transmit a subscore concerning the 



ry 



1 5 essay. 

The present invention is directed to a computer-assisted method of 
evaluating a document concerning at least one topic. The computer-assisted method 
comprises receiving a document from a first computer; electronically comparing textual 
content of the document with predetermined list comprising a first number of terms related 
CO 20 to the at least one topic; identifying terms among the first number which are not present in 
the document; and transmitting the identified terms to the first computer. 



BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is described in detail below with reference to the 
25 drawings in which: 

FIG. 1 shows a block diagram of a grading system in accordance with the 
present invention; 

FIG. 2 shows the process of interacting with an essay grading system in 
accordance with the present invention; 
30 FIG. 3 shows a sample display related to 910 of FIG. 2; 

FIG. 4 shows a sample display related to 912 of FIG. 2; 
FIG. 5 shows another sample display related to 912 of FIG. 2; 
FIG. 6 shows a sample display related to 916 of FIG. 2; 
FIGS. 7a-c show another sample display related to 916 of FIG. 2; 

35 
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FIG. 8 is a flow chart showing the method of the essay grading system 
according to the present invention; 

FIG. 9 shows another sample display related to 916 of FIG. 2; and 
FIG. 10 shows another sample display related to 916 of FIG. 2. 

5 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

FIG. 1 shows a block diagram overview of one embodiment of the essay 
grading system according to the present invention. At least one server computer 1000 is 
connected to a network 1080, as is at least one client computer 1090. The network 1080 
10 can be a local area network (LAN) 1050, such as a network consisting of a plurality of client 
computers 1090 at an examination facility, or a wide area network (WAN) 1060, such as the 
Internet. 

In a preferred embodiment, the server computer 1000 and the client 
computer 1090 are separate computers and may be remotely connected. Alternatively, the 
15 server computer 1000 and the client computer 1090 can be the same computer. In such an 
embodiment, the same computer can possess the capabilities of the server computer 1000 
and those of the client computer 1090. In such circumstances, there may be no need for a 
network. 

The server computer 1000 can include a processor, a disk drive, and a 

20 memory such as a semiconductor memory, and runs an operating system such as 

Windows-NT or Linux. The server computer 1000 is additionally equipped with a data 
communications device such as a network card or gateway to connect to the network 1080. 
In a preferred embodiment, the server computer can also comprise an essay grader 1010, 
implemented in software, and a database 1040. 

25 The database 1040 stores information using, for example, an Oracle 

Relational Database Management System. The information stored in such a database may 
be provided by the entity operating the server. The database, however, can also be used to 
store information sent from, and requests made by, the client computer. In a preferred 
embodiment, the database 1040 can store client information, including client registration 

30 data, requests submitted, results of essay submissions, and scores of submitted essays. The 
database 1040 also stores examination information, including essay questions and a set of 
key terms, or key words, that would be contained in an ideal, model essay associated with 
each essay question. 

The set of key terms that would be contained in an ideal, model essay 

35 associated with an essay question or questions comes from analyses of text books. It may 

-4- DCl -281413.3 



o 

m 

w 



m 

CO 

c:i 



also come from consultation with professors and/or experts in the relevant field. Also, it 
may come from the statistics obtained from the results fed back from professors and/or 
experts in the relevant field in their "trial" examinations conducted prior to the release of 
the question to users in general. A set of key terms, along with the information concerning 
5 these terms, will be associated and designated, and therefore stored together in database 
1040, with each essay question. The key terms are referred to as "ideal terms'*. Variants of 
these terms, such as those formed using common prefixes, suffixes, tenses and the like, are 
either also entered, or are automatically generated on the fly during the analysis of an essay 
or other document. In addition, synonyms of these terms may also be stored, or used by 
10 means of an electronic thesaurus or the like. 

In addition, studies can be conducted, prior to the release of a question, on 
how an essay evaluator would weigh the ideal terms and score a submitted essay based on 
the inclusion and omission of each term within the set of ideal terms. Similar studies can 

tS| also be conducted on how an essay evaluator would grade a submitted essay in terms of a 

Ji 

Jjj 15 general, total score and in terms of subscores. The general, total score concerns the overall 
UJ accomplishment of the submitted essay, such as the overall similarity between the ideal, 

hij model essay and a submitted essay. Each of the subscores concerns a specific aspect of the 

=J| submitted essay. The specific aspects include the content, structure, style, argument, and 

]U specificity concerning a sub-subject. The total score and the subscores may also come from 

CO 20 analyses of text books. They may also come from consultation with professors and/or 
12 experts in the relevant field. Also, they may come from the statistics obtained from the 

C'l results fed back from professors and/or experts in the relevant field in their "trial" 

examinations conducted prior to the release of the question to users in general. One set of 
such evaluation information, including the weighting of the ideal terms and the scoring and 
25 subscoring based on the ideal, model essay, will be associated and designated, and therefore 
stored together in database 1040, with each essay question. This evaluation information is 
referred to as "expert information". 

The essay grader 1010 includes a comparing unit 1020 and a scoring unit 
1030. The comparing unit 1020 comprises software that provides essay analysis functions, 
30 including receiving from a client computer 1090 submissions of an essay associated with a 
question and comparing the submitted essays with a set of ideal terms associated with the 
same question. In a preferred embodiment, the comparing unit 1020 includes software to 
identify the ideal terms that are missing in a submitted essay, based on the ideal terms and 
the associated and designated information stored in database 1040. The ideal terms that are 
35 missing in a submitted essay are referred to as "missed terms". 



-5 - 



DO -281413.3 



Preferably, the comparing unit 1020 also includes software to identify the 
ideal terms that are present in a submitted essay. The ideal terms that are present in a 
submitted essay are referred to as "included terms". 

The scoring unit 1030 contains software which provides essay evaluation 
5 functions, including grading a submitted essay and sending reports back to the client 

computer 1090 concerning the submitted essay. In a preferred embodiment, the scoring unit 
1030 includes software which performs the evaluation of a submitted essay based on the 
expert information stored in database 1040. Preferably, the scoring unit 1030 contains 
software which sends a list of missed terms back to the client computer 1090. Also 
10 preferably, the scoring unit 1030 contains software which sends back to the client computer 
1090 a list of ideal terms, specifying whether each ideal term is a missed term or an 
included term. Such a list is referred to as "verbatim list". 

In general, the connection between the server computer 1000 and the 



J.4 network 1080 can be established in a manner known in the art. For example, the server 

v4 15 computer 1000 can be connected to the Internet via an Internet Service Provider (ISP) or a 
[s| direct connection. In a preferred embodiment, the server computer 1000 is connected to the 

lM Internet via a high-speed connection such as a Digital Subscriber Line or cable connection 

W 

a g or a Tl connection, or the like, in order to receive and service numerous requests at a high 

s data rate. 

0 

j|j 20 In addition, the server computer 1000 is preferably configured to function as 

K| a web server. The web server is typically a general purpose computer such as the server 

computer running software such as Common Gateway Interface (CGI) programs. The CGI 
programs provide for communication and interaction between a client computer 1090 and 



the server computer 1000 via the Internet. These CGI programs, coupled with data 
25 communications software programs, are configured to receive packets of messages from 
computers connected to the Internet, decipher the information in the packets, and act 
according to instructions provided in the packets within the constraints imposed by an 
administrator managing the server computer 1000. Commercial suppliers such as Netscape 
Corporation market web server software. Additionally, such web server software can also 
30 be downloaded and configured free of charge from other sources. Persons skilled in the art 
understand how to write CGI programs in programming languages such as UNIX Shell 
script, Perl, C, C++, Java, and others. 

In addition to performing the tasks of receiving and sending packets of data 
to and from the computers connected to the Internet, the CGI programs are configured to 
35 perform other tasks such as communicating with the database 1040 of the server computer 
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1 000, and extracting or storing information in the database according to the software 
instructions provided within the server computer 1 000 or in the packets received from the 
Internet. The server computer 1000 is configured to receive request messages from client 
computer 1090 over the Internet in the Hyper Text Transfer Protocol (HTTP), File Transfer 

5 Protocol (FTP), or any similar protocol used to transfer data. After analyzing the request 
messages, the server computer 1000 is configured to transmit in response messages that 
include "web pages" that are programmed in Hyper Text Markup Language (HTML) or a 
similar language. Embedded in the web pages are components such as documents, scripts, 
objects, and frames that enable the server computer 1000 to display colorful graphical 

10 images on a display device coupled to the client computer 1090. Persons skilled in the art 
know how to make web pages using programming languages or tools such as HTML, Cold 
Fusion, Java, Java Script, Active Server Pages, Dynamic HTML, Extensible Markup 
Language (XML), etc. 

CIS' 

0 The client computer 1090 is also connected to the network 1080. It may be 
^1 15 connected to a local area network (LAN) 1050, such as a network consisting of a plurality of 
tj| client computers 1090 at an examination facility, or a wide area network (WAN) 1060, such 
W as the Internet. It may be connected to the Internet either directly, or via an Internet service 
„jjf provider, as is known to those skilled in that art. The client computer 1090 may also belong 

1 to a private network having a gateway connection to the Internet. In a preferred 

j|j 20 embodiment, a client computer 1090 illustratively comprises a microprocessor such as a 
M| Pentium EQ microprocessor, a magnetic, optical or other type of disk drive, a memory such 

jjjj as a semiconductor memory, a keyboard or other character input device, a mouse or other 

^ pointing device, and a display device such as a CRT or a flat panel display. In addition to 

operating system software, the client computer 1090 also executes a program called a web 
25 browser, or simply, a browser. The browser is a computer program that provides access to 
the vast resources of the Internet. Typically, this is done by providing a "window" to the 
data located on other computers connected to the Internet. Examples of browser programs 
available are Netscape Navigator, Opera, and Microsoft Internet Explorer. 

FIG. 2 is a flow chart showing the general process when using the essay 
30 grading system according to the present invention. As shown in FIG. 2, the user can log 
into the system at 910. At 912, the user can select essay parameters, which will be 
discussed in detail below, and writes an essay. At 914, the user submits the essay for 
evaluation. Finally, at 914, the user receives and reads the report on the submitted essay. 

FIGS. 3-7 show sample displays of the user interface, preferably displayed at 
35 the user's browser, in a preferred embodiment of the essay grading system. The essay 
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grading system operates in one of two modes: "practice" and "examination". In practice 
mode, the user has a number of options, including the option of essay topic, time, and 
grading policy. The preferred embodiment shown in FIGS. 3-6 is described in practice 
mode. It is readily understood that, in examination mode, the number of options is limited. 

5 FIG. 3 shows a sample display related to 910 of FIG. 2. In this sample, the 

essay grading system transmits a log-in page to the client computer. This page consists of a 
log-in panel 810 with a username field 815, a password field 817, and a login button 819. A 
user, authorized with a username and a password, logs into the essay grading system by 
entering the username in the username field 815, entering the password in the password 

1 0 field 8 1 7, and activating the login button 819. 

FIG. 4 shows a sample display related to 912 of FIG. 2. In this sample, the 
essay grading system transmits an "Essay Options" page to the client computer. This page 
includes 6 panels: an essay style and topic option panel 231, a timing option panel 232, a 
grading policy panel 233, an instruction panel 234, a "Load Old Essay" panel 235, and the 

15 start panel 236. 

The essay style and topic option panel 231 allows an user to choose between 
an SAT II (Scholastic Aptitude Test II) style question and an AP (Advanced Placement) 
style question. Upon choosing a style of question, the user may choose an essay topic, 
which indicates the general area in which a specific essay question will appear for the user 

20 to write on. In other words, the user may choose only the general area the user wishes to 
write about without any control over the exact question the user is to write about. When the 
user selects an appropriate button on the display page of the essay grading system indicating 
an essay style, a list of essay topics is transmitted from the essay grading system server to 
the client computer, and the user may choose an essay topic from the list. 

25 The timing option panel 232 provides two timing options to the user. The 

user can select the "Use default time limit" check box 232a and choose to have a default 
time, which is preferably 30 minutes. Alternatively, the user can select the "Write the essay 
timed" check box 232b and choose the amount of time the user wishes to spend, such as 30 
minutes, 40 minutes, 60 minutes, and so forth. The user can increase or decrease the 

30 amount of time by sliding up or down a slidable scroll 237 in the timing option panel 232. 
The range of the amount of time is between one to 90 minutes and the increment for 
adjustment is one minute. The selected time is displayed in the page and the display of the 
time changes simultaneously as the time is adjusted by the sliding of the slidable scroll. 

The grading policy panel 233 allows the user to choose among three radio 

35 buttons: lenient 233a, normal 233b, and strict 233c. These buttons represent three different 
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grading policies. In the "normal" grading, the submitted essay will be graded against a 
criterion that is used in an actual examination. In the "lenient" grading, the criterion will be 
somewhat lowered and the essay will be graded with leniency. In the "strict" grading, the 
submitted essay will be graded against a higher standard. 

5 The instruction button 234 gives the user the opportunity to obtain 

information concerning the essay grading system. It is similar to a user guide or user 
manual. A user can select on the "Instruction" button 234 for help if the user has a question 
about the essay grading system. 

The "Load Old Essay" button 235 gives a user the option of reviewing an 

10 essay the user submitted earlier. By selecting on the "Load Old Essay" button, the user will 
be prompted for information concerning the old essay, such as the topic and the submission 
date. Upon receiving such information, the essay grading system will execute an 
appropriate software package to verify, from the information stored in the database, whether 
such an old essay exists. If the old essay exists, the essay grading system will display the 

15 old essay and the associated scores and report. That is, when the user selects an appropriate 
button on the display page of the essay grading system indicating the review of an essay 
submitted in an earlier essay session, the essay and the associated scores and report are 
transmitted from the essay grading system server to the client computer. 

Upon a successful selection of essay parameters, the user may select the 

20 "Start Writing Essay" button 236 to cause the essay grading system to go to the next stage 
for the user to start the actual writing. 

FIG. 5 shows another sample display related to 912 of FIG. 2. In this 
sample, the preferred embodiment of the essay grading system provides an essay input 
screen to the user at the client computer. This display contains 9 panels: an essay receiving 

25 panel 259, a status menu 25 1 , a timer panel 252, an essay topic panel 263, an essay question 
panel 254, a "load a different essay topic" button 255, a timing option button 256, and a 
"RocketScore!" button 258 to submit the essay for grading. 

The essay receiving panel 259 provides an editing window for the user to 
input and write an essay in accordance with the essay parameters chosen. 

30 The status panel 25 1 consists of pull-down menus such as "file", "essay", 

"clock", "help" and "debug". These menus enable the user to adequately write in the 
editing window 259, with such editing capabilities as screen copying, pasting, and the like, 
similar to the user would have in a word processor such as WordPerfect and Microsoft 
Word. 
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The timer panel 252 comprises a clock showing the time remaining. While 
the user is writing the essay, the clock ticks away and the time remaining shown in the clock 
reduces preferably a second at a time. For example, if the user chooses the amount of time 
to be 30 minutes in the timing option panel 252 of FIG. 4, the time remaining shown in the 

5 clock will start as 30:00 and progressively reduce to 29:59, 29:58, ... until the user finishes 
the writing and submits the essay by selecting on the "RocketScore" button 258 described 
below, or until the time remaining shown in the clock is reduced to 0:00, when the writing 
is automatically terminated and whatever has been input in the editing window is 
automatically submitted for grading. When the writing is submitted, whether voluntarily or 

10 involuntarily, it is transmitted to the essay grading system for evaluation and scoring. 

The essay topic panel 263 reminds the user of the topic the user chose in the 
essay style and topic option panel 231 of FIG. 4. The essay question panel 254 contains the 
specific question the user is writing about. The question is prompted to the user after the 
user chooses an essay topic, which is an indication of the general area to which the specific 

1 5 essay question belongs. 

The "load a different essay topic" button 255 gives a user the choice of 
changing to another essay topic in a practice session. In a preferred embodiment, a user, 
upon deciding that the current essay topic is not really of interest, may activate the "Load a 
different essay topic" button, whereupon the essay grading system will then loop back to the 

20 window shown in FIG. 4. 

The timing option panel 256 includes the options of "timer enabled" 261, 
"hide timer" 262, and "reset timer" 264. The "timer enabled" check box 261 is useful when 
the user needs to be away for a period of time. Marking the check box 261 enables the 
clock and unmarking the check button disables the clock. When the clock is ticking down 

25 in panel 252 and when the user needs to leave the client computer for a while, the user may 
select the "timer enabled" check box 261 to disable the clock. The user can select the 
"timer enabled" check box 261 again, upon returning to the computer, to enable the clock. 

The "hide timer" check box 262 allows the user to hide the clock in panel 
252. By marking "hide timer" check box 262, the clock will disappear from panel 252 

30 while the time will continue to tick away in the background. By unmarking this check box 
262, the clock will reappear in panel 252. 

The "reset timer" button 264 resets the clock shown in panel 252. It is used 
when the user needs to start the writing over again or when the user needs to adjust the 
amount of time for the writing. It is not available, however, in an examination session. 

35 
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The "RocketScore!" button 258, when activated, submits the writing. After 
the user finishes writing, a user voluntarily submits the writing to the essay grading system 
by selecting the "RocketScore!" button 258. However, when time is up, the essay grading 
system automatically closes the input screen, even if the user has not finished writing, and 

5 transmits whatever is on the screen to the essay grading system for grading without the 
user f s selecting the "RocketScore!" button 258. 

FIG. 6 shows a sample display related to 916 of FIG. 2. In this sample, the 
essay grading system transmits an "essay grader score report" page to the client computer. 
This display includes 9 sections: a score report panel 271, an essay topic panel 272, an essay 

10 question panel 273, a comment panel 274, a panel of missed terms 275, a print button 276, a 
save button 277, a revise button 278, and an "OK" button 279. 

The score report panel 271 includes an overall essay score panel 283, a 
writing subscore panel 281, and content subscore panel 282. Overall essay score panel 
displays the overall score of the submitted writing. It is based on the analysis of the degree 

15 of similarity between the submitted writing and the ideal essay established from the analysis 
of textbooks. Content subscore panel 282 displays a score based on a similarity index 
indicating the similarity between the submitted writing and the ideal essay. It could be 
further broken down to "topic specific" content and "general" content. Writing subscore 
panel 281 displays a score based on how well the user's submission is written. It could be 

20 further broken down to "style" and "structure". 

The essay topic panel 272 recites the essay topic shown in the essay topic 
panel 263 of FIG. 5. The essay question 273 recites the essay question panel 254 of FIG. 5. 
The "note concerning your essay" panel 274 contains some general comments on the writing 
and the panel of missed terms 275 contains a list of missed terms. 

25 The print button 276 allows the user to print out a hard copy of the essay 

grader score report shown in FIG. 6. The save button 277 permits the user to save the essay 
grader score report shown in FIG. 6 in, for example, a file. The revise button 278 allows the 
user to revise the submitted writing by taking the user back to the screen shown in FIG. 5 to 
edit the writing. This option, however, is not available in an examination session. The 

30 "OK" 279 allows the user to end the view of the screen of FIG. 6 and to proceed to the next 
screen display. 

FIGS. 7a-7c show another sample display related to 916 of FIG. 2. In this 
sample, the essay grading system transmits an essay verbatim report display to the client 
computer. This display includes 6 sections: general information 291, overall essay score 
35 292, verbatim list 293, submitted essay 294, resubmission 295, previous performance 296. 
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The sections of general information 291, overall essay score 292, and 
verbatim list 293 are shown in FIG. 7a. General information 291 includes the name of the 
user, the date of the submission of the essay, the subject of the essay, the essay question, and 
the outline topics related to the essay. 
5 Overall essay score 292 recites the overall score and the subscores reported 

on the screen of FIG. 6. It also includes the user's numerical score. 

Verbatim list 293 provides a verbatim list which, as discussed above, is a list 
of the ideal terms, with an indication whether each of these ideal terms is an omitted, or 
missed, term or an included term. In this list, the ideal terms are listed in declining 
10 importance. 

The submitted essay section 294 and resubmission section 295 are shown in 
FIG. 7b. Submitted essay section 294 provides a verbatim display of the submitted essay 
for the user's review. Resubmission section 295 provides instructions on how to edit and 

0 

>J resubmit the essay. The user can follow the instructions and edit and resubmit what the user 

• 3j i 

J;i 15 submitted. 

I ll 

1 1) The previous performance section 296, shown in FIG. 7c, provides a list of 

scores on the user's performance on multiple-choice questions. It shows the topics on which 
the user's performance were poor and suggest that the user avoid these topics when actually 
! rl taking an examination. 

gjj 20 FIG. 8 is a flow chart showing a method of the essay grading system 

according to the present invention. At 210, a user accesses from a client computer a display 
page, such as a web page, transmitted from the essay grading system server to the client 
computer. The user logs on the essay grading system through the display page with a 
username and a password, as discussed earlier in FIG. 3. If it is the first time the user 
25 accesses the essay grading system or if the user does not have ah established account in the 
essay grading system, the user may need to go through a registration process to establish an 
account with a username and a password. Upon successfully logging into the essay grading 
system, the user will be able to access a variety of information on the essay grading system. 
Some of the information may be related to an established account and may be pre-stored in 
30 association with the account. Such information can be automatically activated upon the 
submission of the username and password of the account. 

At 230, the user chooses essay parameters provided by the essay grading 
system. When the user selects an appropriate button on the display page of the essay 
grading system indicating an essay session, the essay parameters are transmitted from the 
35 essay grading system server to the client computer. The essay parameters include a 
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selection of the amount of time to spend, a selection of the grading policy, a selection of 
essay type, and a selection of essay topic, as discussed earlier in FIG. 4. 

At 250, after the user completes the selection of essay parameters, an essay 
question associated with the selected essay topic is extracted from the essay grading system 
5 database and transmitted from the essay grading system server to the client computer. The 
user is also provided a window for inputting an essay in accordance with the essay 
parameters chosen. 

Before submitting the essay, the user has the option of changing essay 
parameters, including the option of changing the topic of the essay, at 253. If the user 
10 decides to make the changes, the essay grading system loops back to 230. Otherwise, if the 
user decides not to change essay parameters, the user may continue writing until the essay 
grading system proceeds to 257. 

At 257, the writing is submitted to the essay grading system for grading. 
^ This can be done in one of two ways. The first way is for the user to activate the 

jjj 15 "Rocketscore" submit button 258 button, once the user is done writing the essay. In this 
iij case, the essay is considered to be voluntarily submitted, since the user determines when to 



submit the essay. The second way is for the system to automatically submit whatever the 
.IS user has entered on panel 259, whether or not the user is finished writing, upon expiration of 

the preset time period. As such, the essay in this second approach, may be considered to 
li| 20 have been involuntarily, since the user takes no affirmative action. This is done by using 
W the clock on the user's computer to keep track of elapsed time. 

P At 260, the essay grading system receives, stores, evaluates and grades the 

submitted essay. The essay grading system compares the submitted essay to a set of ideal 
terms associated with the same essay topic and identifies the omitted, or missed, terms and 
25 the included terms. Also, the essay grading system evaluates the submitted essay. The 
evaluation process includes weighting the ideal terms and calculating the subscores, as well 
as the overall score. This is all done in accordance with the expert information stored in the 
essay grading system. 

At 270, the essay grading system sends feedback to the client computer. The 
30 feedback includes a list of the missed terms, an overall score in numerical representation, 
and subscores in numerical representations, as discussed earlier in FIG. 6. The list of 
missed terms constitutes an in-depth analysis of the submitted essay and serves as feedback 
from which the user can learn and improve the user's score. 

During the review of the feedback, the user is given the option, at 275, of 
35 editing or rewriting the essay. If the user decides to edit or rewrite the essay, the essay 
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grading system loops back to 250. If, on the other hand, the user decides not to edit or 
rewrite the essay, the user can continue reviewing the feedback. 

At 290, the essay grading system transmits an essay verbatim report to the 
user at the client computer. As discussed earlier in FIG. 7a, the essay verbatim report 
5 includes a verbatim list, which contains both the included terms and the missed terms, so 
that the user knows for what terms credit was given, and also which terms should have been 
used. 

Also transmitted from the essay grading system server to the client computer 
is a list of scores on the user f s performance on a set of multiple-choice questions. This list 
10 shows the topics on which the user's performance was poor and provides a form of advice 
for the user to avoid these topics when actually taking an examination. 

Also, during the review of the essay verbatim report, at 310, the user is given 
the option of revising the essay. If the user decides to revise the essay, the essay grading 



\Q system loops back to 250. Otherwise, if the user decides not to revise the essay, the user 



« , 15 can continue reviewing the essay verbatim report. 

I'M 

y At 330, the user is given the option of writing another essay. If the user 

decides to write another essay, the essay grading system loops back to 230. Otherwise, if 
the user decides not to write another essay, the essay grading system proceeds to 350. 
L At 350, the essay grading system updates its database with the information 

C4 2 0 generated in the essay session, including the date and time, the amount of time, the essay 
^ topic, the essay question, the submitted essay, the feedback, and the essay verbatim report. 

CJJ Thereafter, at 370, the essay grading system ends the essay session. 



In another embodiment of the essay grading system, the score report takes 
the form of a pop-up panel 509 with more emphasis on the subscores. FIG. 9 shows a 
25 sample of the pop-up panel, entitled "essay grader score report", which is transmitted to the 
user, upon the user's selecting "RocketScore" 258 in FIG. 5. As shown in FIG. 9, the pop- 
up "essay grader score report" panel 509 includes a score panel 51 1, a comments panel 513, 
a table of contents panel 521, a print entire report button 515, a print this report button 517, 
and an OK button 519. 

30 The table of contents panel 521 lists the contents to be shown in this pop-up 

"essay grader score report" panel. In the sample pop-up "essay grader score report" panel 
509 shown in FIG. 9, the table of contents panel 521 contains a list of first-level categories, 
including "overall score report" 701, "essay information" 715, and "help" 721. The first- 
level category "overall score report" 701 contains a list of second-level categories, including 

35 "writing analysis" 703 and "content analysis" 709. The second-level category "writing 
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analysis" 703 further contains a list of third-level categories, including "discourse subscore" 
705 and "rhetorical subscore" 707 and the second-level category "content analysis" 709 
further contains a list of third-level categories, including "key concepts" 71 1 and "general 
concepts" 713. The first-level category "essay information" 715 contains a list of second- 

5 level categories, including "how to interpret your score" 717 and "essay writing tips" 719. 
When a user selects on a category in the table of contents panel 521, the selected category 
will be highlighted and the contents associated with this category will be shown in the 
appropriate parts of the pop-up "essay grader score report" panel, as discussed below. 

The category "overall score report" 701 is associated with the display of the 

10 total score and the comments concerning the overall performance of the submitted essay. 
The category "writing analysis" 703 is associated with the display of the subscore and 
comments concerning the writing part of the submitted essay. The category "discourse 
subscore" 705 is associated with the display of the subscore and comments concerning the 
discoursing of the submitted essay. The category "rhetorical subscore" 707 is associated 

15 with the display of the subscore and comments concerning the rhetorical aspect of the 
submitted essay. The category "content analysis" 709 is associated with the display of the 
subscore and comments concerning the contents part of the submitted essay. The category 
"key concepts" 71 1 is associated with the display of the subscore and comments concerning 
the key concepts. The category "general concepts" 713 is associated with the display of the 

20 subscore and comments concerning the general concepts. 

The category "essay information" 715 is associated with the display of the 
information concerning the essay writing on the essay grading system. The category "how 
to interpret your score" 717 is associated with the display of the information concerning the 
interpretation of the score and subscores on the essay grading system. The category "essay 

25 writing tips" 719 is associated with the display of tips on essay writing. In addition, the 
category "help" 721 is associated with the display of the technical information related to the 
essay writing on the essay grading system. 

The score panel 511 shows, by default, the overall essay score 527 of a 
submitted essay, in correspondence with the first-level category "overall score report" 701 

30 in the table of contents 521 . It contains the title of "overall essay score" and a number ("9" 
in this instance) showing the overall essay score in numerical representation. The title and 
the number will not change if none of the second- or third-level categories under the first- 
level category "overall score report" 701 in the table of contents 521 is selected. However, 
when a second- or third-level category under the first-level category "overall score report" 

35 701 in the table of contents 521 is selected, the title will change to that associated with the 



$ 

nl 

s s i 
Ml 

et 
q 

•en? 

ri 



15 



DCl -281413.3 



selected second- or third-level category. At the same time, the number will change to that 
associated with the numerical representation of the subscore associated with the selected 
second- or third-level category. 

The comments panel 513 shows a title associated with the selected and 

5 therefore highlighted category in the table of contents 521. It also displays comments that 
are associated with, and change together with, the title. The print entire report button 515 
allows a user to print the entire report, while the print this report button 517 permits the user 
to print the part of the report that is currently shown in the comments panel 513. The OK 
button 519, when activated, closes the pop-up "essay grader score report" panel. 

10 In the screen shot shown in FIG. 9, the category "overall score report" 701 is 

selected and highlighted in the table of contents 521. Accordingly, in score panel 51 1, the 
title is "overall essay score" and the number "9" is the total score. Also, in comments panel 
5 13, the title is "overall essay score" and the comment is "You wrote a strong essay". All 

CJj 

ij the displays in score panel 511 and comments panel 513 are associated with the category 

i;? 15 "overall essay score" selected and highlighted in the table of contents 521 . 

|i| FIG. 10 shows another example of the display of the pop-up "essay grader 

w 
w 
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score report" panel. In FIG. 10, the category "content analysis" is selected and highlighted 
in the table of contents 521. Accordingly, in score panel 51 1, the title is "content analysis 
score" and the number "10" is the subscore of the content analysis aspect of the submitted 
20 essay. Also, comments panel 513 presents a list of missed terms. All the displays in score 
^| panel 511 and comments panel 513 are associated with the category "content analysis" 

C3| selected and highlighted in the table of contents 521. 

While the invention has been described and illustrated herein with respect to 
preferred embodiments, it should be apparent that various alternatives, modifications, 
25 adaptions, and variations will be apparent to those skilled in the art and may be made 
utilizing the teachings of the present disclosure without departing from the scope of the 
invention and are intended to be within the scope of the invention as defined by the claims 
herein. 

30 
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